Encode the residue-only part of a batch in one pass - #124
Merged
Conversation
Encoding is a quarter of the prediction pipeline and every peptide walked through it alone, filling four small arrays a value at a time. The part that depends only on residue identity does not need that: the composition matrix and the one-hot residues are a table gather and a scatter over the batch, the positional block is the same gather at different rows, and the global vector is a sum along the batch's position axis. Bucketed encoding of 20,000 peptides: 0.1837 to 0.0252 ms per peptide, 7.3x, and core.predict on CPU 2,979 to 3,734 peptidoforms/s. encode_peptidoform stays the definition of a feature. Modifications are not reimplemented: they sit on a quarter of a typical peptide list and their placement carries a legacy quirk a shipped model was trained against, so _apply_modifications and its terminal counterpart are called on views into the batch arrays. Exactness comes from reusing that code rather than matching it. Two cases fall back to the per-peptide encoder, both divergences found by testing for them rather than hypotheticals: - Peptides under four residues. The reference's negative positional indices wrap around the sequence - with one residue, seq[seq_len - 2] is seq[-1], that same residue - and the batched form had skipped them. - Peptides carrying a residue the one-hot block has no slot for, such as selenocysteine, whose atoms the reference still reads from pyteomics. Feature layouts the route does not cover at all are declined by supports(): the rolling-sum matrix the four-branch model reads, and the collision cross section extras. Those keep the per-peptide path, as does vectorised_encoding=False on the dataset, which forces it for everything. tests/test_batch_features.py compares the two routes value for value and dtype for dtype over random batches at four windows, truncation, both termini, two modifications, either feature switch, the short and unusual residues above, and through the dataset in both modes. It needs no data files, so CI runs it. Also fixes a crash it surfaced. In _apply_composition_to_matrices the outer try guards KeyError and IndexError, but the inner branch - the one that strips isotope brackets, so C[13] and N[15] as TMT and SILAC labels carry - guarded only KeyError. A labelled modification beyond the padding window therefore raised IndexError where the unlabelled path warned and carried on. 225 tests pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IM2Deep holds a CCS model trained against the pre-4.0.1 encoding and reaches DeepLC through exactly one call, `DeepLCDataset.from_psm_list(psm_list, add_ccs_features=True)`. Two things decline the batched route for that call, the CCS extras and the rolling sum, which `from_psm_list` leaves on. The test asserts the outcome rather than either reason, so removing one of them still fails here. Verified separately end to end: IM2Deep 2.0.2, unmodified, returns bit-identical CCS values on this branch and on the released 4.1.0, over twelve peptidoforms covering side-chain modifications, both termini, an isotope label, a short peptide and selenocysteine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #120. One commit.
Encoding is a quarter of the prediction pipeline and every peptide walked through it alone,
filling four small arrays a value at a time. The part that depends only on residue identity
does not need that.
What it does
the batch;
Bucketed encoding of 20,000 peptides: 0.1837 to 0.0252 ms per peptide, 7.3x, and
core.predicton CPU from 2,979 to 3,734 peptidoforms/s. End to end the encoder is a quarterof the pipeline, so ~1.25x now — and roughly 1.9x once a cheaper trunk makes encoding the
dominant term again.
What it does not do
encode_peptidoformstays the definition of a feature. Modifications are notreimplemented: they sit on a quarter of a typical peptide list and their placement carries a
legacy quirk a shipped model was trained against, so
_apply_modificationsand its terminalcounterpart are called on views into the batch arrays. The exactness comes from reusing
that code rather than from matching it.
Two cases fall back to the per-peptide encoder. Both are divergences I found by testing for
them, not hypotheticals:
the sequence — with one residue,
seq[seq_len - 2]isseq[-1], that same residue — andthe batched form had skipped them.
whose atoms the reference still reads from pyteomics.
Feature layouts the route does not cover at all are declined by
supports(): the rolling-summatrix the four-branch model reads, and the collision cross section extras. Those keep the
per-peptide path, as does
vectorised_encoding=Falseon the dataset, which forces it foreverything. The worst case is therefore no speedup rather than different numbers.
How it is checked
tests/test_batch_features.pycompares the two routes value for value and dtype for dtypeover random batches at four windows, truncation, both termini, two modifications on one
peptide, either feature switch, the short and unusual residues above, and through the dataset
in both modes. It builds its own peptides, so it needs no data files and CI runs it.
Separately verified against the held-out corpus: 20,000 real peptidoforms at windows 20, 30
and 60, byte-identical including dtypes, and identical tensors across every bucket of a real
bucketed prediction.
A crash it surfaced
In
_apply_composition_to_matricesthe outertryguardsKeyErrorandIndexError, butthe inner branch — the one that strips isotope brackets, so
C[13]andN[15]as TMT andSILAC labels carry — guarded only
KeyError. A labelled modification beyond the paddingwindow therefore raised
where the unlabelled path warned and carried on. Reachable today by passing a
padding_lengthbelow your longest peptide. Fixed here with the same guard the sibling branch has, and covered
by a test.
225 tests pass, ruff clean.
🤖 Generated with Claude Code